Diverse Topic Phrase Extraction from Text Collection

Keyword extraction is an efficient approach to managing an explosion of online text on the Web. Traditionally, an abstraction of the online text is constructed though keywords, which are extracted according to a certain importance measure. One such measure is their occurrence frequency. However, previous work has not considered another important factor: the diversity of the keywords. Therefore, the extracted keywords tend to crowd on one hot topic in the corpora while failing to cover other important topics. In this paper, we propose new algorithms to alleviate the disadvantages of these traditional methods for keyword extraction. Firstly, we propose to extract key phrases instead of keywords because phrases can effectively reduce the ambiguity of single words. Secondly, by leveraging latent semantic analysis, we can learn the related topics for each phrase as well as the distance among the phrases, so that the extracted phrases are able to cover more topics. To demonstrate the performance of our method, we conducted experiments on two open datasets: 20 Newsgroup and Reuters-21578.We design three novel evaluation metrics, based on which both qualitative and quantitative analyses shows that our proposed algorithm can be used to improve the key phrase extraction performance significantly.